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1.  Objective 


Unmanned  vehicles  (UV)  are  being  used  more  frequently  in  military  operations,  and  the  types  of 
tasks  they  are  being  used  for  are  evolving  in  complexity.  In  the  future  battlefield,  Soldiers  may 
be  given  multiple  tasks  to  perform  concurrently,  such  as  navigating  a  UV  while  conducting 
surveillance,  maintaining  local  security  and  situational  awareness  (SA),  and  communicating  with 
fellow  team  members.  In  order  to  maximize  human  resources,  it  is  desirable  to  designate  a  single 
operator  to  supervise  multiple  UVs  simultaneously.  However,  past  research  has  shown  that 
human  operators  are  often  unable  to  control  multiple  robots/agents  simultaneously  in  an  effective 
and  efficient  manner  {1,  2).  Additionally,  as  the  size  of  the  robot  team  increases,  the  human 
operators  may  fail  to  maintain  adequate  SA  when  their  attention  has  to  constantly  switch  among 
the  robots,  and  their  cognitive  resources  may  be  overwhelmed  by  intervention  requests  from  the 
robots  {3,  4).  Wang  et  al.  {4)  reviewed  a  number  of  studies  on  supervisory  control  of  multiple 
ground  robots  for  target  detection  tasks  and  concluded  that  “the  fan-out  plateau  lies  somewhere 
between  4  and  9+  robots  depending  on  the  level  of  robot  autonomy  and  environmental  demands” 
(p.  143). 

Research  showed  that  autonomous  cooperation  between  robots  can  aid  the  performance  of  the 
human  operators  (5)  and  enhance  the  overall  human-robot  team  performance  (2).  Wang  et  al.  (4) 
suggest  that  automating  navigation-related  tasks  (e.g.,  path-planning)  is  more  important  than 
“efforts  to  improve  automation  for  target  recognition  and  cueing”  (p.I46)  in  the  context  of 
controlling  a  large  team  of  robots.  However,  in  the  foreseeable  future,  human  operators’ 
involvement  in  mixed-initiative  (i.e.,  human-robot)  teams  will  always  be  required,  especially  for 
critical  decision  making.  Human  operators’  decision  making  may  be  influenced  by  “implicit 
goals”  that  the  robots  are  not  aware  of  (i.e.,  are  not  programmed  into  the  behaviors  of  the  robots) 
(5).  In  addition,  the  real-time  development  on  the  battlefield  may  require  the  human  operator  to 
change  the  plan  for  the  robot  team  and/or  the  individual  robots.  Therefore,  effective 
communication  between  the  human  operator  and  the  robots  is  critical  in  ensuring  mission 
successes.  Research  has  been  conducted  on  ways  to  enhance  human-robot  communication  (6). 
For  example,  researchers  at  Carnegie  Mellon  University  demonstrated  the  effectiveness  of  a 
robot  proxy  to  enhance  shared  understanding  between  the  human  operator  and  the  robot  in  an 
exploration  task  (6).  The  communication  mechanism  was  based  on  a  common  ground 
collaboration  model  and  was  able  to  improve  the  human  operator  performance  in  the  following 
areas:  more  accurate  plans,  more  efficient  planning  (fewer  planning  repetitions),  more  efficient 
and  faster  task  performance,  and  better  mental  model  of  the  capabilities  of  the  robot  (6). 

In  the  current  study,  we  investigated  whether  RoboLeader,  a  robotic  surrogate  for  the  human 
operator  and  an  intelligent  agent  that  can  interpret  the  operator’s  intent  and  issue  detailed 
command  signals  to  a  team  of  robots  of  lower  capabilities,  could  enhance  the  overall  human- 
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robot  teaming  performance.  With  the  RoboLeader  capabilities,  dependence  on  operator 
instructions  was  reduced  and  the  level  of  autonomy  in  operating  UVs  was  improved  by 
implementing  such  algorithms  as  real-time  path  planning,  cooperative  control,  and  multi¬ 
objective  decision  of  tactical  strategies.  These  algorithms  were  stacked  and  the  operator  only 
needed  to  make  high-level  decisions  (7-11).  These  algorithms  resided  in  RoboLeader  and 
enabled  the  operator  to  control  a  team  of  robots  through  a  single  user  interface.  RoboLeader  was 
able  to  assess  the  feasibility  of  the  operator’s  plans  by  simulating  their  execution. 

The  effects  of  individual  differences  factors  on  operator  performance  were  also  evaluated.  More 
specifically,  we  investigated  the  effects  of  individual  differences  in  spatial  ability  (SpA)  and 
perceived  attentional  control  (PAC)  on  the  operators’  robotics  control  as  well  as  multitasking 
performance.  Previous  research  has  demonstrated  that  SpA  is  a  good  predictor  of  the  operator’s 
robotics  performance  (1,  12).  In  the  current  study,  we  also  examined  the  relationship  between 
attentional  control  and  multitasking  performance.  Several  studies  show  that  there  are  individual 
differences  in  multitasking  performance,  and  some  people  are  less  prone  to  performance 
degradation  during  multitasking  conditions  (12).  There  is  evidence  that  people  with  better 
attentional  control  can  allocate  their  attention  more  flexibly  and  effectively  (14),  and  this  was 
partially  confirmed  by  Chen  and  Joyner  (15). 


2.  Approach 


2.1  Participants 

Thirty  individuals  (17  males  and  13  females,  with  a  mean  age  of  24.73  years)  from  the  Orlando, 
FL,  area  participated  in  the  study.  They  were  compensated  $15/h  for  their  time. 

2.2  Apparatus:  Simulator  and  RoboLeader  Algorithm 

The  Mixed  Initiative  Experimental  (MIX)  Testbed  was  modified  and  used  as  the  simulator  (16). 
The  Operator  Control  Unit  (OCU)  of  the  MIX  Testbed  was  modeled  after  the  Tactical  Control 
Unit  developed  under  the  U.S.  Army  Research  Laboratory  (ARL)  Robotics  Collaborative 
Technology  Alliance  (figure  1).  The  path  generator  designed  and  implemented  within  the  MIX 
Testbed  was  designed  for  maximum  search  efficiency,  and  using  simple  concepts  from  vector 
mechanics,  a  unique  behavior  was  given  to  the  planning  characteristics  of  the  path  generator  that 
stands  out  from  typical  matrix  search  algorithms.  Using  vector  mechanics,  the  path  generator  can 
be  given  a  start  point  and  an  endpoint  to  navigate  toward  and  the  algorithm  will  “home  in”  on  the 
intended  destination.  The  path  generator  has  the  ability  to  wrap  around  the  destination  until  an 
entry  path  is  found. 
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Figure  1 .  RoboLeader  interface. 

2,3  Surveys  and  Tests 

A  demographics  questionnaire  was  administered  at  the  beginning  of  the  training  session.  An 
Ishihara  color  vision  test  (with  nine  test  plates)  was  administered  via  PowerPoint®  presentation. 
The  RoboLeader  user  interface  employed  several  colors  to  display  the  plans  for  the  robots  and 
normal  color  vision  was  required  to  effectively  interact  with  the  system.  A  questionnaire  on 
Attentional  Control  {14)  was  used  to  evaluate  participants’  perceived  attentional  control.  The 
Attentional  Control  survey  consists  of  21  items  and  measures  attention  focus  and  shifting.  The 
scale  has  been  shown  to  have  good  internal  reliability  (a  =  0.88).  The  Cube  Comparison  Test 
(77)  and  Spatial  Orientation  Test  {18)  were  used  to  assess  participants’  SpA.  The  Cube 
Comparison  Test  required  participants  to  compare,  in  3  min,  21  pairs  of  6-sided  cubes  and 
determine  if  the  rotated  cubes  were  the  same  or  different.  The  Spatial  Orientation  Test,  modeled 
after  the  cardinal  direction  test  developed  by  Gugerty  and  his  colleagues  {18),  is  a  computerized 
test  consisting  of  a  brief  training  segment  and  32  test  questions.  Both  accuracy  and  response  time 
were  automatically  captured  by  the  program.  Participants’  perceived  workload  was  evaluated 
using  the  computerized  version  of  National  Aeronautics  and  Space  Administration  Task  Load 
Index  (NASA-TLX)  questionnaire,  using  a  pairwise  comparison  weighting  procedure  {19).  The 
NASA-TLX  is  a  self-reported  questionnaire  of  perceived  demands  in  six  areas;  mental,  physical, 
temporal,  effort  (mental  and  physical),  frustration,  and  performance.  Participants  evaluated  their 
perceived  workload  level  in  these  areas  on  10-point  scales  as  well  as  completed  pairwise 
comparisons  for  each  subscale. 
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2.4  Procedure 

Participants  were  randomly  assigned  to  the  RoboLeader  group  or  the  baseline  (no  RoboLeader) 
group.  After  the  informed  eonsent  form  proeess,  partieipants  eompleted  the  demographie 
questionnaire  and  the  Attentional  Control  survey,  and  were  administered  a  brief  Ishihara  eolor 
vision  test  and  two  spatial  ability  tests.  After  the  spatial  tests,  partieipants  reeeived  training, 
whieh  lasted  about  1  h  and  was  self-paeed,  delivered  by  PowerPoint  slides  showing  the  elements 
of  the  OCU  user  interface,  steps  for  eompleting  various  tasks,  and  several  mini-exercises  for 
praetieing  the  steps.  Eaeh  experimental  session  had  two  scenarios,  each  lasting  approximately 
30  min,  in  whieh  partieipants  used  their  robotie  assets  to  loeate  10  targets  (i.e.,  insurgents 
carrying  weapons)  in  the  remote  environment.  There  were  four  robots  available  in  one  scenario 
and  eight  robots  in  the  other  seenario.  The  order  of  seenarios  was  eounterbalaneed  aeross 
participants.  The  experimental  session  lasted  about  1  h. 

When  eaeh  scenario  started,  the  robots  began  by  following  pre-planned  routes,  at  whieh  time  the 
operator’s  task  of  monitoring  the  environment  and  deteeting  insurgents/impro vised  explosive 
deviees  (lEDs)  began.  The  robots  did  not  have  Aided  Target  Recognition  (AiTR)  capability; 
therefore,  the  partieipants  had  to  deteet  the  10  insurgents  and  10  lEDs  themselves.  Eor  the 
insurgents,  participants  were  instrueted  to  use  their  eomputer  mouse  to  elick  on  the  targets  to 
“lase”  them  as  soon  as  they  were  detected.  The  “lased”  insurgents  were  automatieally  displayed 
on  the  map.  Eor  the  lEDs,  the  partieipants  elieked  on  the  lED  button  on  the  interfaee,  and  then 
marked  the  loeations  of  the  lEDs  on  the  map.  Additionally,  there  were  friendly  dismounted 
Soldiers  and  eivilians  in  the  simulated  environment  to  inerease  the  visual  noise  for  the  target 
deteetion  tasks.  The  partieipants  were  told  that  their  objeetive  was  to  finish  reeonnoitering  the 
area  using  their  robotic  assets  in  the  least  amount  of  time  possible.  Therefore,  when  re-planning  a 
route,  the  participant  and/or  RoboEeader  must  consider  both  the  effeetiveness  and  effieiency  of 
the  new  route.  Situations  where  a  robot  eompleted  its  route  quiekly  but  did  not  eover  mueh 
ground,  or  where  the  robot  eovered  a  lot  of  ground  but  was  slow  to  finish,  were  eonsidered 
suboptimal  in  eomparison  to  re-planning  a  route  that  effieiently  (i.e.,  less  time)  eovered  a  lot  of 
ground.  RoboLeader  would  reeommend  new  routes  for  robots  that  finished  first,  if  it  deeided  that 
the  overall  mission  time  could  be  reduced  by  redireeting  those  robots  to  the  unsearched  areas. 

In  eaeh  scenario,  there  were  six  events  that  required  revisions  to  a  robot’s  eurrent  plans/route. 
Once  an  event  transpired,  the  baseline  partieipants  had  to  notiee  that  the  event  had  oeeurred  (via 
an  auditory  alert)  and  then  re-route  the  robot  that  was  affeeted  by  the  event.  Eor  those  in  the 
RoboLeader  eondition,  the  RoboLeader  would  reeommend  plan  revisions  to  the  operator,  who 
eould  either  aeeept  the  plans  or  modify  them  as  neeessary.  Out  of  these  six  events,  three  were 
“bottom-up”  (e.g.,  unantieipated  obstaeles  deteeted  by  the  robots  that  obstruet  their  navigation) 
and  three  “top-down”  (e.g.,  intel  that  the  human  operator  reeeives  from  the  intel  network).  Given 
that  the  events  led  to  obstruetions  (e.g.,  vehieles  in  the  path,  hostile  area),  the  RoboLeader  and 
the  partieipants  needed  to  avoid  re-routing  through  these  areas,  in  addition  to  avoiding  areas 
where  insurgents  or  lEDs  were  already  detected. 
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In  each  scenario,  there  were  five  SA  queries,  which  were  triggered  based  on  time  progression 
(e.g.,  3  min  into  the  scenario).  The  SA  queries  included  questions  such  as  “which  areas  have  the 
robots  searched?”  (the  participant  marked  the  searched  areas  on  a  blank  map),  “which  of  your 
robots  is  the  closest  to  [Area  of  Interest],”  etc.  The  OCU  screen  was  blank  when  an  SA  query 
was  triggered,  and  only  the  SA  queries  were  displayed  on  the  screen. 

There  were  2-min  breaks  between  experimental  sessions.  Participants  assessed  their  perceived 
workload  (NASA-TLX)  after  each  experimental  scenario. 

The  study  was  a  mixed  design,  with  RoboLeader  (with  or  without  RoboLeader  [baseline])  as  the 
between-subject  variable,  and  the  number  of  robots  used  in  the  scenario  (four  vs.  eight)  the 
within-subject  variable.  Performance  measures  included  number  of  targets  located  and  identified, 
the  operator’s  SA  of  the  mission  environment,  as  well  as  awareness  of  the  status  of  the  individual 
robots.  A  mixed-design  analysis  of  covariance  (ANCOVA)  with  RoboLeader  (with  or  without 
RoboLeader)  as  the  between-subject  factor  and  number  of  robots  (four  vs.  eight)  as  the  within- 
subject  factor  was  used  to  evaluate  the  operator’s  performance  differences  among  the  four 
conditions.  Participants’  SpA  (composite  score  of  the  two  spatial  tests)  and  their  attentional 
control  survey  scores  were  used  as  covariates. 


3.  Results 


3,1  Target  Detection  Performance 

The  analysis  revealed  that  the  robots’  condition  significantly  affected  the  number  of  targets 
detected,  F(l,26)  =  2\.12,p  <  0.0001.  Participants  detected  significantly  fewer  insurgents  when 
they  had  eight  robots  compared  with  the  condition  when  four  robots  were  available.  Participants 
with  a  higher  SpA  detected  significantly  more  insurgents  than  those  with  a  lower  SpA  did, 
F(l,26)  =  6.63,  p  <  0.05  (figure  2).  The  effects  of  RoboLeader  and  attentional  control  were  not 
statistically  significant.  The  analysis  also  showed  that  participants  detected  significantly  fewer 
lEDs  when  they  had  eight  robots  compared  with  the  condition  when  four  robots  were  available, 
F(l,26)  =  \6.\3,p  <  0.005.  Participants  with  a  higher  SpA  detected  significantly  more  insurgents 
than  those  with  a  lower  SpA  did,  F(l,26)  =  10.66, <  0.005  (figure  3).  The  effects  of 
RoboLeader  and  attentional  control  were  not  statistically  significant.  There  was  a  non-significant 
RoboLeader  x  SpA  effect,  F(l,26)  =  2.55,/>  =  0.12.  As  can  be  seen  in  figure  3,  participants  with 
a  higher  SpA  performed  slightly  better  with  RoboLeader  compared  with  the  baseline  condition; 
however,  the  same  pattern  was  not  observed  for  the  lower  SpA  group. 
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Figure  2.  Insurgent  detection  performance. 


Figure  3.  lED  detection  performance. 

3,2  SA  Queries 

The  analysis  revealed  that  participants’  SA  was  significantly  lower  when  they  had  eight  robots 
compared  with  the  condition  when  four  robots  were  available,  F(l,26)  =  13.3 1, />  <  0.005.  None 
of  the  other  factors  were  significant  (figure  4). 
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Figure  4.  Situation  awareness. 


3,3  Perceived  Workload 


The  analysis  showed  that  partieipants  experieneed  signifieantly  higher  workload  when  there 
were  eight  robots  {M=  69.3)  vs.  four  robots  (M=  64.3),  F(l,26)  =  4.95,/?  <  0.05  (figure  5). 
Participants  in  the  RoboLeader  group  assessed  their  workload  slightly  lower  {M=  64.1 1)  than 
those  in  the  baseline  group  (M=  69.4)  did.  However,  the  difference  failed  to  reach  statistical 
significance.  Participants  with  higher  attentional  control  rated  their  workload  as  significantly 
lower  than  those  with  lower  attentional  control  did,  F(l,26)  =  7.23,/?  <  0.05.  Notably,  females 
reported  significantly  higher  workload  (4-robot:  74.97;  8-robot:  77.05)  than  males  (4-robot: 
56.18;  8-robot:  63.29)  did,  F(l,28)  =  12.16,/?  <  0.005. 
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Figure  5.  Perceived  workload. 


7 


3,4  Operators’  Interaction  with  the  OCU 

Participants’  interaction  with  the  OCU  (e.g.,  elieks  on  the  graphieal  user  interfaee  [GUI])  was 
further  analyzed.  Overall,  participants  in  the  RoboLeader  spent  significantly  less  time 
eompleting  their  mission  scenarios  than  did  those  in  the  baseline  group,  F(l,27)  =  l.\2,p<  0.05. 
Participants  in  the  RoboLeader  group  spent,  on  average,  20.7  min  per  scenario;  while,  those  in 
the  baseline  group  spent  23.8  min  per  scenario.  During  the  experiment,  participants  needed  to 
eliek  on  the  smaller  thumbnails  (i.e.,  streaming  videos  from  the  robots)  to  enlarge  the  video 
image  in  order  to  identify  targets.  Participants  in  the  baseline  group  made  significantly  more 
clicks  on  the  thumbnails  than  those  in  the  RoboLeader  group  did,  F(l,25)  =  8.33,/?  <  0.01.  They 
also  made  more  elieks  when  they  had  eight  robots  eompared  with  the  four-robot  eondition, 
F(l,25)  =  132.23,/?  <  0.001.  Additionally,  there  was  a  signifieant  RoboLeader  x  Robots 
interaction,  F(l,25)  =  5.09,/?  <  0.05  (figure  6).  The  difference  between  the  baseline  and  the 
RoboLeader  groups  was  greater  in  the  eight-robot  condition  than  that  in  the  four-robot  eondition. 


Figure  6.  Thumbnail  clicks  per  minute. 


4.  Conclusions 


An  intelligent  agent,  RoboLeader,  was  developed  to  assist  operators  in  eontrolling  a  team  of 
robots  in  route  planning  tasks.  Although  no  signifieant  differences  were  found  in  target  detection 
between  the  groups,  the  RoboLeader  group  completed  their  missions  in  significantly  less  time 
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than  those  in  the  baseline  group.  On  average,  RoboLeader  saved  the  partieipants  '-'3  min  for 
missions  lasting  20+  min.  While  this  finding  was  expected  given  the  assistance  from 
RoboLeader  in  route  planning  tasks,  participants  in  the  RoboLeader  group  exhibited  more 
pronounced  complacency  in  the  eight-robot  condition  compared  to  the  baseline  group.  More 
specifically,  the  RoboLeader  participants  made  fewer  clicks  on  thumbnails  of  the  streaming 
video  from  the  robots  (to  look  for  targets)  in  the  eight-robot  condition.  While  the  number  of 
clicks  did  not  directly  translate  into  higher  number  of  targets  detected,  the  striking  difference  in 
the  way  the  operators  interacted  with  the  OCU  based  on  the  presence  of  RoboLeader  requires 
further  investigation. 

We  compared  the  operators’  target  detection  performance  in  four-  and  eight-robot  conditions. 
The  results  showed  that  the  participants  detected  significantly  fewer  insurgents  and  lEDs  when 
there  were  eight  robots  compared  to  the  four-robot  condition,  indicating  less  effectiveness  with 
more  resources/assets.  Those  participants  with  higher  SpA  detected  more  insurgents  as  well  as 
lEDs.  These  results  are  consistent  with  previous  findings  that  individuals  with  a  higher  SpA  tend 
to  exhibit  more  effective  scanning  performance  and,  therefore,  are  able  to  detect  more  targets 
than  individuals  with  a  lower  SpA  {1,  15).  It  is  likely  that  RoboEeader’s  utility  was  not  sufficient 
to  overcome  the  effect  of  SpA.  In  other  words,  the  participants  with  higher  SpA  were  able  to 
outperform  those  with  lower  SpA,  regardless  of  the  RoboEeader  condition. 

When  there  were  eight  robots,  the  participants’  SA  was  significantly  worse  than  when  there  were 
four  robots.  On  the  other  hand,  the  SA  of  the  RoboEeader  participants  was  not  significantly 
degraded  compared  with  the  baseline  group.  In  other  words,  the  out-of-the-loop  phenomenon 
associated  with  automation  (75)  was  not  manifest  in  the  RoboEeader  condition. 

Einally,  participants  experienced  significantly  higher  workload  when  there  were  eight  robots 
compared  to  the  four-robot  condition,  and  those  with  better  attentional  control  reported  lower 
workload  than  did  those  with  poorer  attentional  control.  Eemales  also  reported  significantly 
higher  workload  than  males  did.  Those  participants  in  the  RoboEeader  group  rated  their 
workload  as  slightly  lower  than  those  in  the  baseline  group  did,  although  the  difference  did  not 
reach  statistical  significance. 
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6.  Transitions 


RoboLeader  will  be  used  to  support  our  year  two  Director’s  Research  Initiative  (DRI)  research  in 
which  the  capabilities  of  RoboLeader  will  be  expanded  to  deal  more  specifically  with  dynamic 
re-tasking  requirements  for  persistent  surveillance  of  a  simulated  urban  environment  based  on 
various  battlefield  developments  (e.g.,  individual  robots  need  to  be  re -tasked  to  search  for  a  high- 
stakes  target).  Furthermore,  the  capability  of  RoboLeader  will  be  extended  beyond  coordination 
with  homogeneous  assets  (i.e.,  unmanned  ground  vehicles  [UGVs])  to  coordination  with 
heterogeneous  assets  (i.e.,  unmanned  aerial  vehicles  [UAVs]  and  UGVs)  when  in  pursuit  of 
moving  targets  in  urban  environments.  Additionally,  RoboLeader  will  be  used  for  studies  of 
automation  reliability  and  operator  individual  differences  in  future  Safe  Operations  for 
Unmanned  Reconnaissance  in  Complex  Environments  (SOURCE)  Army  Technology  Objective 
(ATO)  research. 

A  technical  report  is  being  prepared,  as  well  as  papers  for  submission  at  conferences  on  robotics 
and  human-robot  interaction. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


AiTR  Aided  Target  Recognition 

ANCOVA  analysis  of  covariance 

ARL  U.S.  Army  Research  Laboratory 

ATO  Army  Technology  Objective 

DRI  Director’s  Research  Initiative 

GUI  graphical  user  interface 

lEDs  improvised  explosive  devices 

MIX  Mixed  Initiative  Experimental 

NASA-TEX  National  Aeronautics  and  Space  Administration  Task  Eoad  Index 

OCU  Operator  Control  Unit 

PAC  perceived  attentional  control 

SA  situational  awareness 

SOURCE  Safe  Operations  for  Unmanned  Reconnaissance  in  Complex  Environments 

SpA  spatial  ability 

UAVs  unmanned  aerial  vehicles 

UGVs  unmanned  ground  vehicles 


UV  unmanned  vehicles 
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